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ABSTRACT 

Technical  Report 

This  paper  presents  the  results  of  a  survey  of  companies  involved  in  the 
development  and  supply  of  complex  operational  computer  based  systems  for 
Navy.  The  objective  of  the  survey  was  to  gain  feedback  from  industry 
regarding  the  quality  of  Navy's  requirements  specifications.  The  survey  is  part 
of  a  more  general  study  aimed  at  improving  Navy's  specification  and  evaluation 
practices. 
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1.  Introduction 

This  paper  presents  the  results  of  a  survey  of  companies  involved  in  the 
development  and  supply  of  complex  operational  computer  based  systems  for 
Navy.  The  objective  of  the  survey  was  to  gain  feedback  from  industry 
regarding  the  quality  of  Navy's  requirements  specifications.  The  survey  is  part 
of  a  more  general  study  aim^  at  improving  Navy's  specification  and  evaluation 


practices. 

The  full  series  is  as  follows: 

Report  1:  Industry  survey 

Report  2;  Current  policy 
and  practice 

Report  3:  Requirements  and 
specifications 

Report  4:  Executive 
summary  and  final 
recommendations 


A  survey  of  industry  perceptions  of 
Navy  specifications. 

Examines  the  current  policy  and  practice 
regarding  the  development  of 
specifications  in  Navy.  (Not  Public 
Release.) 

A  comprehensive  review  of  the  needs  for 
Navy  specifications,  providing 
recommendations  for  their 
improvement. 

Consolidated  recommendations  arising 
from  Reports  1, 2  and  3.  (Not  Public 
Release.) 


The  specifications  under  review  are  those  provided  as  part  of  a  Request  for 
Proposal  (RFP)  or  Request  for  Tender  (RFT).  In  general,  these  specifications 
contain  performance  and  functional  requirements  as  well  as  build  standard 
requirements.  They  are  based  on,  and  traceable  to,  higher  level  operational 
requirements  prepared  by  end  user  representatives.  The  higher  level 
requirements  documents  are  not  normally  provided  to  potential  tenderers. 

Surveys  of  this  type  provide  an  unavoidably  jaundiced  view  of  the  subject. 
Participants  were  encouraged  to  criticise  specification  practices  and  did  so.  As  a 
result  the  majority  of  the  comments  were  negative  and  this  report  reflects  those 
views.  Where  possible,  however,  we  have  also  documented  positive  comments 
indicating  practices  preferred  by  the  participants. 

While  this  paper  discusses  problems  with  Navy  specifications,  several 
participants  expressed  the  view  that  Navy  specifications  are  often  superior  to 
other  Defence  specifications,  particularly  in  providing  a  clear  and  reasonably 
high  level  representation  of  the  requirements.  In  addition,  few  of  the  comments 
applied  to  all  specifications  produced  by  Navy,  and  most  of  the  comments  were 
applicable  in  some  measure  to  other  Defence  specifications.  In  this  regard,  this 
paper  is  generally  relevant  to  all  Defence  specifications  for  operational  systems. 
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A  recent  survey  into  the  costs  of  tendering  for  Defence  projects  resulted  in 
several  comments  relevant  to  this  survey.  These  are  discussed  in  Section  16. 

Finally,  it  should  be  noted  that,  although  some  comments  are  made  on  the 
participants’  suggestions,  the  aim  of  this  paper  is  to  document  industry 
perceptions.  While  it  Is  Important  that  their  views  are  considered,  tenderers  are 
only  one  of  several  important  audiences  for  specifications.  Accordingly,  this 
paper  should  not  be  viewed  either  as  a  recipe  for  specifications  or  a  blueprint  for 
future  changes.  A  traceability  matrix  is  provided  in  Appendix  2  showing  where 
Issues  raised  here  are  addressed  in  Reports  2  and  3. 


2.  Acknowledgments 

The  authors  wish  to  acknowledge  the  cooperation  and  assistance  of  staff  in  the 
following  organisations  who  participated  in  the  survey. 

Atlas  Elektronik 

Australian  Defence  Industries 

Australian  Submarine  Corporation 

AWA  Defence  Industries 

Blohm  and  Voss 

British  Aerospace  Australia 

CEA 

CelsiusTech  Australia 
Computer  Sciences  Corporation 
GEC-Marconi  Systems 
Rockwell  Systems  Australia 
Stanilite 

Thomson  Sintra  Pacific 
Transfield  Shipbuilding 


3.  The  survey 

In  all,  14  companies  participated  in  the  survey.  These  constituted  the  majority 
of  Australian  companies  which  might  be  expected  to  tender  for  combat  systems 
and  other  major  operational  computer  based  systems  in  Australia.  Some  of  the 
systems  addressed  are  shown  below. 
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ANZAC  Frigate  combat  system  and  ship  control  system 
Hydrographic  Ship  science  suite 

Collins  Submarine  combat  system  and  ship  control  system 
Minehunter  Coastal  combat  system 
Minewarfare  Systems  Centre 
Patrol  Vessel  combat  system 

There  was  also  some  discussion  of  specifications  from  Defence  clients  other  than 
Navy,  including  Army,  Air  Force  and  Defence  Materiel.  In  total,  more  than  20 
separate  projects  were  discussed  in  the  interviews.  Discussions  typically 
covered  most  of  the  projects  that  contractors  had  tendered  for,  and  were  not 
restricted  to  projects  in  which  they  had  been  awarded  the  contract. 

The  survey  was  conducted  by  structured  interviews,  with  the  industry 
participants  being  provided  with  a  list  of  subjects  to  be  addressed  prior  to  the 
interviews.  The  questionnaire  used  is  shown  in  Appendix  1.  The  subjects 
addressed  included  the  following: 

Content  and  completeness 

Level  of  requirements 

Structure,  format  and  language 

Non-functional  requirements 

Other  specific  concerns  including  testability,  prioritisation  and 
security  issues 

Electronic  form  of  specifications 
Pre-release  of  specifications  to  tenderers 
Industry  specification  practices 

There  was  a  high  level  of  cooperation  from  participants  resulting  in  extremely 
frank  comments  at  times.  This  was  supported  by  guarantees  of  confidentiality 
on  our  part,  so  that  specific  comments  are  not  attributed  to  individuals  or 
companies. 


4.  Content  and  completeness 

The  content  of  a  specification  here  refers  to  the  appropriateness  of  requirements 
rather  than  the  level  of  detail  provided  in  the  specification  or  the  manner  in 
which  the  requirements  are  presented  (which  are  addressed  elsewhere  in  this 
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paper).  Completeness  refers  to  whether  the  full  requirement,  as  envisioned  by 
the  potential  tenderer,  is  encompassed. 

4.1  Assumed  knowledge 

Tenderers  have  difficulties  in  imderstanding  some  requirements  which  appear 
to  be  inconsistent  or  poorly  matched  to  the  bulk  of  the  specification.  This 
problem  is  often  attributed  to  the  drafter  of  the  specification  using  assumptions 
which  are  unknown  to  the  tenderer. 

4.2  "Cut  and  paste"  approach 

Requirements  are  sometimes  copied  from  previous  projects  with  limited 
consideration  of  whether  they  are  appropriate  for  the  system  being  specified. 
Specific  concerns  with  this  practice  included  the  following. 

a.  Requirements  which  are  suitable  for  one  project  will  not  necessarily  be 
suitable  for  another.  The  roles  of  two  ships  may  be  different,  for 
example,  and  the  required  performance  of  their  systems  will  usually  be 
different. 

b.  Requirements  copied  from  an  older  project  may  not  reflect  advances  in 
technology  and  system  performance.  This  is  discussed  further  in  Section 
4.4. 

4.3  "Glossy  brochure"  approach 

There  were  suggestions  from  participants  that  some  specifications  are  based  on 
promotional  material  ("glossy  brochures")  from  defence  suppliers,  rather  than 
genuine  requirements,  although  it  was  also  stated  that  this  problem  has 
improved  in  recent  years.  The  concerns  raised  included  the  following. 

a.  Figures  provided  in  promotional  material  are  often  the  best  performance 
that  can  be  achieved  under  optimal  environmental  conditions.  They  are 
almost  universally  more  optimistic  than  the  typical  performance  which 
might  be  expected  from  systems,  or  the  performance  that  a  tenderer  can 
guarantee  in  a  contract.  Use  of  such  figures  will  ensure  that  no  tenderer 
can  be  compliant. 

b.  Where  several  of  the  performance  characteristics  of  a  specific  system  are 
echoed  in  a  specification,  the  specification  may  reflect  the  actual  design 
of  that  system  rather  than  Navy's  higher  level  requirements.  Competing 
systems  will  have  a  different  set  of  characteristics  which,  while  they  may 
meet  the  higher  level  requirements,  cannot  match  the  specified 
requirements  point  by  point. 

c.  It  was  suggested  that  in  some  cases  the  requirements  were  based  on  a 
combination  of  performance  figures  from  different  brochures. 
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In  these  cases,  it  is  evident  either  that  no  existing  equipment  will  meet  Navy’s 
requirements,  or  at  best  that  only  one  set  of  equipment  will  be  compliant.  The 
likely  consequences  are  that  competition  is  decreased  and  that  there  is  less 
likelihood  that  existing  equipment  will  meet  the  requirements. 

4.4  Technology  constraints 

Some  specifications  have  included  requirements  which  preclude  the  use  of  more 
advanced  technology.  Tenderers  have  then  been  reluctant  to  offer  solutions 
which  would  have  provided  enhanced  performance  and  reduced  costs,  but 
which  would  be  formally  non-compliant. 

Most  participants  recommended  that  particular  care  should  be  taken  when 
requirements  relate  to  immature,  emergent  or  rapidly  changing  technologies, 
and  that  in  these  cases  requirements  should  be  stated  at  as  high  a  level  as  is 
prudent.  This  is  particularly  relevant  for  long  projects  where  technology 
advances  during  the  course  of  the  project  may  make  some  detailed 
requirements  (such  as  requirements  for  computer  types  and  capacities)  either 
obsolete  or  difficult  to  achieve. 

4.5  "Umbrella”  clauses 

Requirements  which  apply  to  the  entire  system  or  to  several  components  can 
cause  problems  for  a  tenderer  where,  in  the  tenderer's  opinion  at  least,  the 
requirements  are  inappropriate  for  some  system  components.  This  occurs 
frequently  with  regard  to  non-functional  requirements  including  build 
standards.  A  common  example  is  "All  software  shall  be  developed  in 
accordance  with  IX)D-STD-2167A,  Defense  System  Software  Development”.  The 
tenderer  knows  than  many  of  the  proposed  system  components  which  contain 
software  (often  in  the  form  of  firmware)  already  exist,  but  is  uncertain  about  the 
status  of  that  software.  He  is  faced  with  the  following  problems: 

•  Does  the  requirement  apply  to  tried  and  tested  firmware  in  items 
such  as  monitors  and  printers? 

•  Does  it  apply  to  existing  software  in  subsystems  such  as  radars? 

•  Does  it  apply  to  existing  mission  specific  software  or  only  to  new 
software? 

•  Should  any  existing  software  be  redeveloped  or  redocumented  to 
conform  with  the  standard? 

The  tenderer  cannot  answer  these  questions  in  isolation,  and  requires  further 
guidance  from  the  customer. 

4.6  Environmental  constraints 

Some  tenderers  felt  that  the  operational  environment  is  not  defined  in  sufficient 
detail.  This  applies  both  to  external  environmental  factors  (such  as  sea  state, 
wind  and  temperature)  and  to  the  environment  in  which  the  equipment  is 
installed,  e.g.  within  a  ship. 
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4.7  Traceability  of  operational  to  performance/functional 
requirements 

Several  participants  commented  on  the  lack  of  clear  traceability  between 
operational  requirements,  where  provided,  and  corresponding  performance  and 
functional  requirements.  It  was  suggested  that  some  of  the  lower  level 
requirements  were  either  "created  from  nothing"  or  were  based  on  operational 
requirements  which  had  not  been  identified  in  the  specification  (see  also  Section 
4.1).  This  has  resulted  in  a  difficulty  in  understanding  the  context  of 
requirements  and  a  subsequent  difficulty  in  responding  to  them. 

4.8  Use  of  standards 

There  was  considerable  concern  about  the  referencing  of  military  specifications 
and  other  standards.  Typical  problems  included  the  following. 

a.  Referenced  standards  may  be  inappropriate  for  the  application. 

b.  Standards  listed  as  applicable  documents  are  sometimes  not  referenced 
in  the  text  of  the  specification,  raising  doubt  as  to  their  applicability. 

c.  Some  standards  (eg  DOD-STD-2167A)  require  specific  decisions  to  be 
made  or  actions  to  be  performed  by  the  customer,  but  the  customer  does 
not  appear  to  be  aware  of  these  obligations. 

d.  Many  standards  are  multi-level  providing  for  different  levels  of 
compliance  depending  on  the  needs  of  the  application.  In  some  cases 
the  actual  level  required  is  not  stated. 

e.  The  use  of  "orphan"  standards  which  are  not  widely  used  either  in 
defence  or  industry  can  seriously  impact  cost  and  schedule. 

f.  Many  standards  have  not  been  updated  with  advances  in  technology  or 
changes  in  the  ADO  organisation.  The  ABR  and  DI(N)  series  were 
specifically  mentioned  in  this  regard.  It  was  suggested  that  a 
comprehensive  review  process  to  upgrade  or  retire  such  standards 
would  significantly  reduce  this  problem. 

g.  Some  referenced  standards  (e.g.  STANAGs)  are  difficult  or  impossible  to 
obtain. 

Most  tenderers  would  welcome  both  the  tailoring  of  standards  by  Navy  for 
specific  projects  and  the  provision  of  guidance  on  the  applicability  and  use  of 
referenced  standards. 

4.9  Completeness  of  requirements 

There  were  no  criticisms  of  the  completeness  of  specifications.  While  this  might 
be  seen  to  be  a  tribute  to  specification  drafters,  it  is  more  likely  due  to  the  fact 
that  missing  requirements  do  not  usually  result  in  problems  for  the  contractor. 
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Instead,  their  impact  is  borne  by  Navy  both  in  performance  and  cost. 
Participants  also  felt  that  the  channels  available  to  resolve  such  problems  (with 
the  Project  Office)  generally  worked  well. 


5.  Level  of  requirements 

5.1  Uneven  level  in  specifications 

All  participants  commented  on  the  uneven  level  of  individual  specifications. 
WhUe  it  was  clear  that  some  requirements  had  been  specified  at  too  high  a  level, 
and  others  too  low,  there  was  rarely  complete  agreement  between  participants 
with  regard  to  specific  problems.  Problems  cited  included: 

a.  Different  drafting  styles  in  the  same  specification,  particularly  in 
different  system  areas  (e.g.  the  Combat  System  compared  with  the 
Communications  System). 

b.  Evident  variability  in  the  experience  of  drafters. 

c.  Copying  from  earlier  specifications  which  use  quite  different  formats 
and  drafting  styles. 

d.  Mixing  of  genuine  requirements  with  partial  solutions,  which  may  lead 
to  serious  problems  in  designing  a  cost  effective  solution. 

e.  Areas  of  overspecified  requirements,  including  identification  of 
computer  types  and  computer  languages,  specification  of  solutions 
rather  than  requirements,  detailed  requirements  which  assume  a 
particular  solution,  and  specification  of  inappropriate  processes  and 
protocols. 

f.  Areas  of  underspecified  requirements,  including  the  operating 
environment,  maintenance,  simulation  for  training,  administrative 
computing  and  fault  tolerance. 

There  was  acceptance  from  most  participants  that  some  variability  in  the  level  of 
specification  is  appropriate  and  unavoidable.  One  example  occvirs  where  the 
interfaces  of  a  system  to  existing  equipment  must  be  strictly  defined,  but  other 
aspects  of  the  system  can  be  specified  solely  in  terms  of  performance  and 
functions.  The  level  is  likely  to  vary  with  both  how  detailed  the  users’ 
requirements  need  to  be  de^ed  and  how  detailed  they  can  be  defined.  This  is 
discussed  further  below. 

It  was  also  accepted  that  in  some  cases  functional  requirements  will  be 
incomplete;  that  only  a  subset  of  the  required  functions  will  be  specified.  In 
these  cases  it  was  recommended  that  the  higher  level  requirements  are  clearly 
identified,  so  that  the  complete  set  of  functional  requirements  can  be  derived  by 
the  tenderer. 
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5.2  Vague  requirements 

The  use  of  requirements  which  express  a  genuine  need,  but  in  a  form  which  is 
too  general  to  be  completely  understood,  concerned  participants.  Requirements 
such  as  "shall  maintain  the  existing  capability",  "shall  be  compatible  with 
current  Navy  protocols"  and  "shall  be  a  fully  integrated  system"  were  seen  as 
being  very  difficult  to  address.  It  was  suggested  that  either  the  requirement 
should  be  clarified,  or  that  guidance  should  be  provided  on  how  tenderers 
should  respond  to  the  requirement. 

5.3  What  is  the  correct  level? 

The  variation  of  responses  to  our  questions  with  regard  to  level  was  interesting. 
The  discrepancies  applied  to  individual  specifications,  where  there  were 
different  views  on  the  quality  of  a  given  specification,  and  also  specific 
technology  areas,  such  as  communications. 

Most  participants  agreed  that  there  is  no  "correct"  level  of  specification  which 
will  be  appropriate  for  all  projects  or  suit  all  tenderers.  Tenderers  with 
extensive  experience  in  a  particular  domain  prefer  high  level  specifications, 
whereas  the  same  tenderers  will  prefer  detailed  requirements  in  areas  where 
their  experience  is  less.  Similarly,  specification  drafters  depend  heavily  on  their 
own  and  Navy's  experience  both  in  the  application  domain  and  technologies 
involved.  It  appears  that  most  problems  in  level  of  detail,  both  too  high  and  too 
low,  occur  when  the  drafter  has  insufficient  or  inappropriate  experience.  The 
most  serious  problems  occur  when  both  Navy  and  the  tenderer  have  a  low  level 
of  experience. 

Participants  were  particularly  scathing  about  inappropriate  or  incorrect  (in  their 
opinion)  overspecification,  which  they  believed  had  precluded  cost  effective 
solutions. 

Cases  were  also  cited  where  the  level  of  specification  was  "exactly  wrong"; 
where  for  example  specific  tools  were  required  but  no  indication  was  provided 
on  why  the  tools  were  selected  or  how  they  would  be  used  to  meet  the  higher 
level  requirements.  In  these  cases  tenderers  would  have  found  either  a  higher 
or  lower  level  acceptable. 

One  interesting  outcome  of  the  survey  was  an  almost  universal  opinion  among 
the  participants  that  all  specifications  contain  areas  where  the  requirement  is 
inadequately  specified,  i.e.  where  more  detail  is  needed  to  guide  the  tenderer. 
This  contradicts  a  common  belief  that  the  problems  are  solely  in  the  other 
direction,  that  overspecification  is  by  far  the  major  problem.  Individual 
concerns  with  high  level  specifications  included  the  following. 

a.  There  is  a  significant  risk  that  the  customer  will  not  agree  with  the 
derived  requirements,  and  will  reject  the  solution. 

b.  The  tenderer  needs  to  make  too  many  guesses. 
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c.  If  Navy  cannot  specify  its  requirements,  neither  can  the  tenderer. 

d.  Acceptance  of  systems  with  complex  user  interfaces  becomes  very 
difficult  and  a  serious  area  of  risk  to  the  contractor. 

e.  High  level  performance  specifications  are  incompatible  with  fixed  price 
contracts. 


6.  Structure  and  format 

Structure  here  refers  to  the  (usually)  methodical  approach  to  organisation  of  the 
requirements  in  the  specification.  Format  refers  to  the  layout  of  the  specification 
document,  which  can  also  dictate  or  constrain  the  structure. 

Many  of  the  suggestions  relating  to  structure  and  format  stemmed  from 
difficulties  in  importing  specifications  into  requirements  databases.  Most 
tenderers  parse  specifications  into  database  or  dedicated  requirements 
traceability  tools.  The  objective  of  the  parsing  process  is  to  identify  and  classify 
each  discrete  requirement  in  the  RFT/RFP  package  (including  implicit 
requirements  in  some  cases)  to  assist  in  ensuring  that  the  tender  or  proposal 
addresses  all  requirements.  Various  characteristics  of  the  specification  can  assist 
in  or  detract  from  the  parsing  process. 

6.1  Use  of  defined  formats 

There  was  general  concern  about  the  variability  in  structure  and  format  of  the 
specifications  experienced.  Ship  specifications  have  usually  been  based  on  the 
GENSPECS  (General  Specifications  for  Ships  of  the  United  States  Navy) 
standard,  using  Technical  Subject  Codes  (TSCs)  to  partition  the  requirements. 
System  specifications  are  often  loosely  based  on  MIL-STD-490A  (even  when  the 
ship  specification  uses  GENSPECS).  Most  participants  recommended  the  use  of 
the  490A  System  Specification  format  -  almost  all  participants  use  strict  490A 
formats  for  their  own  subcontract  specifications. 

The  reasons  given  for  preferring  MIL-STD-490A  were  that  this  standard 
provides  a  predictable  framework  for  the  more  common  requirement  criteria,  is 
more  likely  to  encourage  completeness  of  specification,  and  is  reasonably  well 
understood  in  the  industry. 

Several  participants  criticised  the  use  of  GENSPECS  and  490A  formats  for 
integrated  systems,  because  of  the  difficulties  in  representing  integrated 
requirements  in  a  partitioned  form  (using  TSCs)  or  a  strictly  hierarchical  form 
(for  490A).  It  was  generally  agreed  that  the  structure  of  the  actual  performance 
requirements  should  be  appropriate  to  the  system  being  specified,  rather  than 
follow  the  directives  of  490A,  for  example.  It  was  felt  that  the  use  of  TSCs  in 
particular  encouraged  drafters  to  design  systems  to  the  equipment  level  rather 
than  to  concentrate  on  the  establishment  of  performance  and  functional 
requirements. 
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6.2  Individual  format  issues 

Problems  in  parsing  specifications  into  discrete  requirements  prompted  many 
suggestions  regarding  the  layout  of  specifications.  These  comments  included 
the  following. 

a.  Each  clause  should  contain  only  one  requirement.  While  this  is  often 
quoted  as  a  good  objective,  there  is  often  disagreement  between  different 
parties  as  to  what  coirstitutes  a  single  requirement.  This  will  also  vary 
when  tenderers  have  different  specific  solutions  or  a  conceptual  design 
in  mind.  If  a  single  requirement  is  met  by  a  single  component  or 
equipment,  there  will  generally  be  no  problem  if  the  requirement  has 
two  or  more  components. 

b.  Avoid  using  tables  to  identify  requirements.  Tables  are  difficult  to  parse 
using  automated  parsing  tools,  but  are  a  useful  way  of  presenting  large 
numbers  of  simple  requirements  such  as  data  items. 

c.  Avoid  using  lists  (such  as  this  one)  in  requirements,  because  each 
individual  requirement  may  not  make  sense  without  the  preceding  text 
at  the  start  of  the  list. 

d.  Avoid  "overlap"  requirements  where  the  requirements  for  one  section 
refer  to  the  requirements  in  another,  with  specific  additions  or 
exclusions.  An  example  of  such  a  requirement  is  "all  data  in  paragraph 
4.17  shall  be  recorded,  with  the  exception  of  operator  errors  and  BITE 
data".  Again,  the  requirement  is  not  complete  in  itself.  In  addition, 
participants  identified  such  requirements  as  likely  candidates  for 
inconsistencies  and  suspect  requirements  (requirements  which  the 
tenderer  believes  are  not  genuinely  intended). 

e.  Minimise  "umbrella"  clauses  where  the  scope  of  a  requirement  applies  to 
aU  or  several  system  components. 

f.  Collect  like  requirements  together  or  provide  cross-references  where  this 
is  not  feasible.  An  example  of  this  is  data  recording  requirements  where 
the  requirements  for  recording  might  be  scattered  through  the 
specification  in  different  functional  areas.  It  will  not  always  be  possible 
to  group  related  requirements  in  integrated  systems  because  a  single 
function  will  often  be  provided  by  several  system  components.  In  the 
data  recording  example,  the  requirements  for  the  recording  of 
navigational  data,  for  instance,  could  be  specified  as  part  of  the 
navigation  requirements  or  as  part  of  the  recording  requirements. 

Many  of  these  suggestions,  if  implemented  strictly,  could  drastically  reduce  the 
readability  of  specifications,  which  in  turn  would  adversely  affect  the  validation 
of  requirements  and  the  ability  of  readers  in  general  to  assimilate  the  meaning 
of  specifications.  A  compromise  is  evidently  needed  here,  where  the 
requirements  for  parsing  are  traded  off  against  the  needs  of  readability. 
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For  example,  the  "one  requirement  per  clause"  rule  is  now  usually  followed  in 
Navy,  but  not  to  the  point  of  absurdity.  There  is  often  a  difference  of  opinion  on 
what  constitutes  a  single  requirement.  Examples  of  Navy  specifications  parsed 
by  tenderers  and  sighted  by  the  authors  have  shown  both  the  merging  and 
splitting  of  requirements,  with  different  approaches  taken  by  different 
tenderers.  Similarly,  participants  were  divided  in  their  opinions  of  how  well 
requirements  were  partitioned  in  the  same  specifications. 

6.3  Numbering  of  requirements 

Several  participants  believed  that  the  assignment  by  Navy  of  a  unique  number 
to  each  requirement  (in  effect  each  "shall")  would  be  very  useful.  This  would 
reduce  the  probability  of  a  contractor  "losing"  a  requirement  during  parsing  and 
aid  in  cross  referencing  requirements  between  different  specifications.  The 
approach  used  in  the  MWSC  project,  where  sub-requirements  in  each  clause 
were  numbered  ([1],  [2],  etc.),  was  generally  seen  as  adequate;  in  this  case  a 
requirement  could  be  referred  to  as  3.7.1. 4[3],  for  example.  In  other 
specifications  requirements  have  been  numbered  using  a  sequential  numbering 
scheme  (in  addition  to  and  separate  from  clause  numbers). 

It  is  important  to  note  here  that  the  ordering  of  numbers  is  not  seen  as 
important,  nor  is  the  fact  that  some  numbers  in  the  sequence  may  not  be  used 
(after  requirements  are  deleted,  for  example).  The  important  issue  is  that  each 
requirement  may  be  uniquely  referred  to,  and  that  the  identifying  number  does 
not  change  after  the  first  release  of  the  specification. 

The  generation  and  maintenance  of  such  numbers  may  be  difficult  to  manage 
without  the  likelihood  of  duplications,  omissions  and  errors.  This  problem  will 
be  examined  in  a  subsequent  report. 

6.4  Precedence  of  specifications,  standards  and  requirements 

The  precedence  of  different  specifications,  standards  and  requirements  needs  to 
be  carefully  considered  and  defined  where  there  is  any  Ukelihood  of  overlap.  It 
should  be  noted  that  the  precedence  of  requirements  is  not  the  same  as  the 
priority  of  different  requirements  which  is  discussed  in  Section  8. 

6.5  Dispersed  requirements 

Associated  requirements  should  be  addressed  in  a  single  specification  where 
possible.  Participants  quoted  projects  where  technical  requirements  relating  to 
a  common  area  appeared  in  the  specification,  the  RFT  Terms  and  Conditions, 
the  Data  Item  Descriptions  and/or  the  Statement  of  Work.  (The  coUection  of 
like  requirements  within  a  specification  is  addressed  in  Section  6.2.f.) 
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7.  Verification  of  requirements 

Most  participants  stated  that  all  requirements  should  either  be  testable,  or  an 
indication  should  be  provided  on  how  they  will  be  evaluated.  In  some  cases  the 
level  of  capability  required  had  not  been  clear  to  those  preparing  the  technical 
proposal,  and  it  was  felt  that  some  indication  of  how  the  capability  would  be 
verified  would  help  to  clarify  the  requirement.  Where  the  method  of 
verification  (which  includes  testing)  is  not  obvious,  it  was  recommended  that 
the  specification  should  provide  guidance  on  what  verification  methods  might 
be  used  in  system  acceptance. 

It  should  be  noted  that  for  a  contractual  specification,  all  requirements  need  to  be 
subject  to  verification,  usually  by  testing,  inspection  or  analysis. 


8.  Prioritisation  of  requirements 

Many  participants  felt  that  a  clear  prioritisation  of  requirements  would  improve 
their  ability  to  apply  cost  benefit  analysis  and  criticality  analysis  techniques  in 
Navy  projects.  It  was  suggested  that  prioritisation  to  3  levels  -  mandatory, 
highly  desirable  and  desirable  -  would  be  a  marked  improvement  on  current 
practices.  Clear  definition  of  operational  requirements,  including  the 
operational  concept,  would  help  the  tenderer  to  assess  priorities. 

There  appears  to  be  a  suspicion  that  some  requirements  are  more  speculative 
than  real,  and  that  these  will  not  be  treated  seriously  in  the  tender  evaluation 
process.  Participants  also  stated  that  in  some  cases  they  get  a  different 
impression  of  the  relative  importance  of  requirements  in  discussions  with  staff 
from  different  areas  in  Navy.  This  latter  problem,  although  a  concern,  is 
perhaps  understandable  -  specialists  tend  to  regard  their  own  part  of  a  project 
as  the  most  important.  Any  solution  to  this  problem  should  address  the 
promotion  of  a  firm  official  position  with  regard  to  priorities  rather  than  reduce 
the  already  limited  dialogue  between  the  tenderers  and  specialists. 

In  most  specifications  the  prioritisation  is  limited  to  mandatory  ("shall")  and 
non-mandatory  ("should"  or  desirable)  requirements.  There  were  indications 
during  the  interviews  that  tenderers  have  difficulty  in  addressing 
non-mandatory  requirements,  because  of  uncertainty  in  how  these 
requirements  are  treated  in  evaluations.  This  is  also  shown  in  tender  responses 
which  are  highly  variable  in  their  treatment  of  non-mandatory  requirements. 


9.  Use  of  language 

The  use  of  language  is  not  generally  regarded  as  a  cause  of  serious  problems. 
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9.1  Definition  and  use  of  terms 

There  were  some  concerns  about  the  use  of  specialised  operational  terms  which 
are  not  adequately  defined  or  which  are  used  inconsistently.  The  different 
meaning  of  such  terms  in  the  military  in  Australia,  the  US  and  Europe  has  also 
led  to  some  problems. 

9.2  Concurrency 

Whether  functions  are  required  concurrently  or  not  is  often  not  clearly  specified. 
This  often  results  in  problems  when  the  contractor  designs  the  functions  to  be 
mutually  exclusive  but  the  customer  assumes  that  both  (or  all)  functions  will  be 
available  simultaneously. 

9.3  Alternative  functions 

Where  alternative  functions  are  specified,  it  is  not  always  clear  whether  the 
choice  is  being  offered  to  the  tenderer  or  the  operator.  An  example  might  be 
"Positions  shall  be  entered  either  graphically  or  numerically  using  the 
keyboard".  This  probably  means  that  the  operator  needs  two  alternative 
methods  of  entering  position,  but  it  could  also  be  interpreted  as  meaning  that 
the  designer  has  a  choice  of  providing  either  graphical  or  numeric  entry, 
without  providing  both.  The  use  of  "or"  in  a  requirements  clause  can  quite  often 
be  ambiguous. 


10.  Non-functional  requirements 

Requirements  which  do  not  relate  directly  to  the  performance  or  functions  of  a 
system  are  often  referred  to  as  non-functional  requirements.  Typical 
non-functional  requirements  are  the  "ilities"  (availability,  maintainability, 
compatibility  and  other  quality  factors),  buUd  standards  and  user  interfaces.  It 
shoiild  be  noted  that,  despite  their  name,  these  requirements  usually  do 
contribute  to  the  overall  effectiveness  and/ or  efficiency  of  a  system. 

Non-functional  requirements  are  viewed  as  a  serious  area  of  risk.  Most 
participants  stated  that  they  are  typically  poorly  defined,  difficult  to  respond  to, 
and  in  many  cases  difficult  to  verify  (they  are  often  not  easy  to  test  or  measure). 
There  were  also  suggestions  that  this  area  is  particularly  prone  to 
mistmderstandings  in  the  use  of  special  terms.  Specific  comments  concerning 
these  requirements  were  as  follows. 

a.  Non-functional  requirements  should  be  clearly  derived  from  the 
operational  requirement  where  possible. 

b.  AU  non-functional  requirements  need  to  be  considered  and  specified. 
The  use  of  a  comprehensive  checklist  by  Navy  would  assist  in  this 
regard. 
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c.  In  some  cases  there  are  direct  conflicts  between  different  requirements. 
One  example  is  where  the  requirements  for  availability  and  MTTR 
(Mean  Time  to  Repair)  cannot  be  reconciled. 

d.  There  is  little  consideration  given  to  the  quality  factors  for  shore  based 
facilities. 

e.  Specification  of  requirements  for  the  user  interfaces  is  seen  to  be  a  critical 
problem  area.  Trying  to  meet  the  subjective  views  of  customers  is  seen 
as  a  serious  risk. 

f.  Customers  need  to  understand  that  verification  of  non-functional 
requirements  in  particular  may  involve  more  than,  or  exclude,  testing. 
Alternative  methods  may  include  analysis,  inspection  and  comparison 
with  existing  systems. 


11.  Security  issues 

Tenderers  stressed  the  costs  and  time  delays  in  handling  classified  requirements 
and  documents,  and  believe  that  there  are  often  cases  of  overclassification  which 
makes  these  penalties  unnecessary.  Other  comments  included  the  following. 

a.  Portion  marking  (individual  paragraph  classification)  is  strongly 
preferred  to  blanket  document  classification. 

b.  Separation  of  classified  from  unclassified  material  using  different 
documents  is  preferred  to  documents  of  mixed  classification,  particularly 
where  there  are  relatively  few  classified  requirements.  The  use  of 
classified  annexes,  particularly  for  the  extraction  of  performance  figures 
("magic  numbers"),  was  recommended. 


12.  Electronic  form  of  specifications 

Navy  specifications  are  now  routinely  provided  to  tenderers  in  electronic  form 
as  wordprocessor  dociunents  or  databases.  Tenderers  strongly  support  this 
practice  and  consider  it  essential  that  all  RFT/RFP  information  is  provided  in 
electronic  form.  Participants  commented  on  the  variability  in  both  the  format  of 
the  specifications  and  how  the  generation  tools  are  used.  (This  has  been 
aggravated  by  the  recent  change  of  the  wordprocessor  type  generally  used 
within  Navy.) 

Understandably,  participants  preferred  the  use  of  popular  brand  tools,  and 
some  were  strongly  critical  of  Filemaker  Pro  (database  tool)  and  Wingz 
(spreadsheet)  in  this  regard.  Many  stated  that  they  would  prefer  to 
immediately  import  such  material  into  their  database  of  choice,  rather  than  use 
a  tool  which  they  are  unfamiliar  with  or  which  is  inadequate  for  their  purposes. 
They  recommended  that  the  specification  be  provided  both  as  generated,  and  in 
a  popular  compatibility  format  suitable  for  importation  into  other  tools. 
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Similarly,  where  tender  responses  are  required  in  a  specific  format,  it  was 
suggested  that  a  popular  compatibility  format  should  also  be  acceptable.  (An 
example  of  a  popular  compatibility  format  for  databases  is  comma  separated 
variable  (CSV)  which  is  computer  independent  and  suitable  for  import  to  and 
export  from  all  modem  database  tools.) 

It  was  also  suggested  that  Navy  defines  and  applies  standards  formalising  how 
the  tools  are  used  within  Navy.  The  use  of  consistent  wordprocessor  formatting 
using  styles,  for  example,  not  only  enhances  the  consistency  of  the  specification 
format,  but  also  can  be  very  useful  in  parsing  for  requirements  management  by 
the  tenderer.  Seemingly  simple  issues  such  as  the  inclusion  of  hard  page  breaks 
and  the  arbitrary  or  manual  formatting  of  paragraphs  and  headings  can  cause 
problems,  particularly  when  the  tenderer  needs  to  process  several  versions  of  a 
specification. 

It  was  also  noted  that  the  use  of  graphics  in  specifications  can  cause  problems  in 
electronic  handling,  particularly  in  increasing  the  size  of  the  file,  and  in 
compatibility  between  different  tools.  It  was  suggested  that  graphics  should  not 
be  explicitly  included  in  the  specification  file,  but  instead  be  linked  into  the 
document  from  separate  individual  files.  (This  is  possible  in  most  modem 
wordprocessing  tools  including  Microsoft  Word  and  WordPerfect.) 


13.  Pre-release  of  specifications 

All  participants  supported  the  pre-release  of  draft  specifications,  provided  for 
information  or  for  comment.  Some  felt  that  the  opportunity  to  comment  on  the 
draft  specification  was  useful,  to  modify  the  requirement  and  improve  their 
ability  to  submit  a  compliant  tender.  They  also  welcomed  the  chance  to  discuss 
the  requirement  informally  with  Navy  project  staff,  specialists  and  users,  to  give 
them  a  better  imderstanding  of  the  requirement,  an  activity  which  is  not 
possible  later  in  the  procurement  process.  Others  believed  that  commenting  on 
the  specification  might  result  in  changes  that  helped  other  tenderers  and  were 
more  cautious  in  doing  so.  Some  expressed  concern  that  by  asking  basic 
questions  relating  to  the  requirements,  they  might  create  an  impression  of  not 
being  conversant  with  the  application  domain,  and  that  this  might  also  tend  to 
reduce  the  feedback  they  might  provide  to  the  project. 

All  saw  the  increase  in  their  lead  time  for  the  preparation  of  proposals  as  being 
one  major  advantage  of  their  viewing  a  draft  version  of  the  specification. 


14.  Miscellaneous  issues 


14.1  Additional  information  -  need  for  operational  requirements 

All  tenderers  desire  more  information  relating  to  the  operational  requirements, 
which  they  believe  would  significantly  improve  their  ability  to  respond  to  the 
requirements.  Most  have  had  difficulty  in  resolving  how  some  individual 
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performance  and  functional  requirements  might  have  been  derived  and  attempt 
to  postulate  the  operational  requirements  themselves  using  reverse  engineering, 
often  with  a  low  level  of  confidence.  In  most  cases  the  information  required 
does  not  need  to  be  a  formal  part  of  the  requirements,  and  could  be  provided  as 
a  preface  or  annex  to  the  specification. 

Several  participants  stated  that  they  would  welcome  any  additional  information 
relating  to  the  requirements  that  can  be  provided.  One  example  of  additional 
information  cited  was  annotations  or  notes  clarifying,  justifpng  or  commenting 
on  the  requirements. 

14.2  Overall  specification  quality 

Several  participants  suggested  that  many  of  the  problems  in  specifications 
might  be  averted  if  Navy  could  establish  generic  quality  guidelines  for  their 
specifications,  and  review  each  specification  against  those  guidelines  prior  to 
release. 

14.3  Relationship  with  customer 

Some  participants  indicated  that  they  would  prefer  a  much  closer  relationship 
with  the  customer.  They  felt  that  the  contractual  relationship  as  currently 
conducted  prevents  or  distorts  important  discussions  between  the  supplier  and 
the  customer  relating  to  the  overall  requirements. 

14.4  Extraneous  issues 

Our  discussions  were  primarily  aimed  at  Navy's  specifications,  but  numerous 
other  issues  were  rais^  which  participants  thought  detracted  from  their  abUity 
to  respond  adequately  to  RFT/RFPs.  These  are  not  discussed  in  this  report. 
Several  participants  believed  that  there  are  more  serious  problems  in  other 
aspects  of  the  tendering  process  than  in  the  actual  specifications.  (One  example 
which  was  raised  by  several  participants  was  the  amount  and  format  of  tender 
documentation  deliverables  required  by  projects.) 


15.  Industry  specification  practices 

We  were  interested  in  how  different  tenderers'  specification  practices  differ  from 
Navy's,  particularly  because  they  are  based  on  the  same  original  requirements. 
In  many  cases,  where  a  major  system  is  subcontracted,  as  is  likely  when  a 
subcontractor  supplies  a  combat  system  for  a  ship.  Navy's  requirements  are 
generally  passed  on  virtually  vmchanged  to  the  subcontractor,  with  additional 
(typically  more  stringent)  environmental  and  build  standard  requirements. 

For  smaller  systems  and  system  components  two  quite  different  approaches 
appear  to  be  common,  depending  on  the  products  required  and  the  prime 
contractor's  experience  with  the  technologies  or  application  involved. 
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a.  Provide  a  product  or  product  development  specification  with  detailed 
requirements,  generally  at  a  level  lower  than  those  used  by  Navy.  The 
reason  given  for  the  prime  contractor  using  a  low  level  specification  was 
that  they  know  what  they  want,  include  as  much  information  as  they 
can,  and  see  a  detailed  specification  as  the  best  method  for  achieving 
their  objective. 

b.  Provide  a  short,  high  level  description  of  requirements,  and  award  a 
contract  on  the  basis  of  a  product  or  product  development  specification. 

Almost  all  of  the  pcU'ticipants  use  a  MIL-STD-490A  format  (or  OOD-STD-2167A 
for  software)  as  a  basis  for  specification  for  subcontractors.  The  reasons  given 
for  preferring  MIL-STD-490A  were  that  this  standard  provides  predictable 
hooks  for  the  more  common  requirement  criteria,  is  reasonably  complete  and  is 
reasonably  well  understood  in  the  industry. 

Several  participants  stated  that  their  own  RFTs  provide  more  detailed  guidance 
than  Navy  on  how  the  tenderer  should  respond  to  the  requirements,  and  in 
some  cases  a  specification  blank  is  included  for  the  tenderer  to  complete. 

Most  do  not  currently  provide  explicit  numbering  of  requirements  (discussed  in 
Section  6.3),  but  intend  to  do  so  in  the  future. 


16.  Costs  of  tendering  survey 

Recently  a  survey  was  commissioned  to  review  Defence  procurement  practices, 
particularly  with  regards  to  costs  of  industry  tendering  for  Defence  projects. 
The  results  of  the  survey,  which  encompassed  80  firms,  were  published  in  Costs 
of  Tendering  Industry  Survey,  Australian  Government  Publishing  Service, 
Onberra,  1994.  It  should  be  noted  that  the  survey  encompass^  all  Defence 
procurement  and  was  not  selective  about  the  types  of  projects  covered 
(although  48%  of  respondents  were  from  the  information  technology, 
electronics  and  communications  sector).  Participants  were  those  doing 
"significant  amounts  of  business  (i.e.  contracts  of  $1  million  or  more)  with 
Defence". 

Many  of  the  responses  and  comments  provided  in  the  survey  are  relevant  to 
specifications  and  reinforce  the  results  of  our  survey.  These  included: 

•  Poor  requirements  specifications  which  vary  during  the  tendering 
process. 

•  Requirements  written  around  dated  technology. 

•  Overly  prescriptive  specifications. 

•  The  large  numbers  of  standards  to  be  obtained. 

•  An  "arms  length"  approach  requiring  the  tenderer  to  guess 
requirements. 

•  The  need  for  clear  specification  of  requirements. 

•  Preference  for  fewer  optional  requirements. 
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The  Costs  of  Tendering  survey  specifically  requested  comments  on  the  value  of 
circulating  draft  specifications  to  tenderers,  prior  to  calling  for  tenders.  Around 
50%  of  the  respondents  favoured  this  approach,  while  all  participants  in  our 
survey  favoured  pre-release  of  specifications.  It  is  possible  that  this  difference  is 
due  to  the  large  size  and  complexity  of  the  projects  we  were  investigating,  when 
compared  to  the  more  general  scope  of  the  Costs  of  Tendering  survey. 
Individual  comments  provided  on  the  pre-release  of  specifications  were  almost 
identical  to  those  in  our  survey,  however,  and  included  the  following: 

•  Circulation  of  draft  specifications  provides  more  time  for  the 
tenderer. 

•  Commercial  advantages  and  disadvantages  of  providing  feedback. 

•  Allows  early  decision  on  whether  to  tender. 

•  Increases  time  and  effort  required  to  tender. 

•  Can  be  confusing  if  there  are  significant  changes  to  the  specification. 


17.  Conclusions 

This  survey  has  provided  a  valuable  insight  into  industry’s  perceptions  of  Navy 
specifications  for  complex  computer  based  systems.  It  has  identified  many  of 
the  perceived  weaknesses  of  specification  practices,  as  well  as  some  of  the 
strengths,  and  has  provided  a  sound  basis  for  the  investigation  and 
improvement  of  those  practices  in  the  future. 
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Appendix  1  -  Survey  Questionnaire 

The  foUowing  questionnaire  was  sent  to  participants  prior  to  interviews  and 
were  used  as  the  basis  for  discussions. 

Background 

We  are  currently  tasked  by  Navy  to  review  Navy's  specification  and  evaluation 
practices  for  complex  operational  computer  bas^  systems  (such  as  combat 
systems).  We  expect  that  the  results  will  have  a  wider  impact,  both  in  Navy 
and  Defence. 

As  part  of  this  task  we  are  conducting  a  survey  within  both  Navy  and  defence 
industry  to  investigate  the  perceived  problems  in  Navy  specifications. 

It  is  likely  that  the  findings  of  this  task  will  be  releasable  to  the  defence 
community,  at  least  in  part. 

The  survey 

Attached  is  a  list  of  questions  which  we  are  using  as  the  basis  for  discussions 
with  defence  industry  in  Australia.  Although  phrased  as  a  series  of  questions,  it 
is  not  intended  that  participants  will  complete  the  questionnaire  themselves, 
although  you  may  chose  to  do  so.  We  would  prefer  to  discuss  these  and  other 
issues  with  you,  using  the  questionnaire  to  guide  the  discussions. 

Individual  answers  to  these  questions  will  not  be  formally  recorded  or 
published  and  will  be  treated  confidentially.  This  is  essential  to  increase  the 
frankness  and  honesty  of  answers. 

Participants  are  encouraged  to  address  specific  projects  and  specifications  rather 
than  provide  a  general  response  from  their  experience. 

Questionnaire 

Where  possible,  please  address  your  answers  to  specific  projects  and 
specifications.  The  specifications  of  interest  are  generally  those  provided  as  part 
of  an  RFP  or  RFT  process. 

1.  What  Navy  projects/ specifications  have  you  recently  experienced? 

2.  What  was  your  general  impression  of  the  specification  in  comparison 
with  those  for  other  projects? 

3.  What  specific  strengths  did  you  perceive  in  the  structure,  content  and 
language  of  the  specification? 

4.  What  specific  problems  were  in  your  opinion  caused  by  the  structure, 
content  and  language  of  the  specification? 
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5.  Do  you  have  any  comments  about  the  level  of  specification,  particularly 
writh  regard  to  pure  performance  based  specifications  (which  do  not 
define  functional  requirements)? 

6.  Do  you  have  any  comments  on  the  completeness  of  specification? 

Would  you  have  appreciated  more  information? 

7.  Oo  you  have  any  suggestions  with  regard  to  the  specification  of  non¬ 
functional  requirements  (including  quality  factors)? 

8.  What  advantages  do  you  see  in  viewing  a  pre-release  specification  and 
being  able  to  provide  conunents? 

9.  What  factors  cause  difficulties  in  understanding  and  using  Navy 
specifications? 

10.  What  changes  to  specifications  or  to  the  specification  process  would 
improve  yoiu  ability  to  understand  Navy  specifications  and  to  respond 
to  them? 

1 1 .  How  do  these  specifications  differ  from  those  that  you  supply  to  your 
own  subcontractors? 

12.  Do  you  use  any  specific  tools  to  analyse  or  break  down  specifications  for 
further  use  (eg  requirements  tracking)? 

13.  What  are  your  preferences  with  regard  to  the  electronic  form  of 
specifications? 

14.  Any  other  comments? 
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Appendix  2  -  Traceability  Matrix 


The  following  table  provides  traceability  of  issues  identified  in  this  report  to  the 
sections  where  these  issues  are  addressed  in  subsequent  reports. 


Section 

Issue 

See  Report:Section 

4.1 

Assumptions  by  drafters 

3:6.2.1 

rmm 

"Cut  and  paste"  from  previous  projects 

tarn 

"Gtossy  brochure"  approach 

3:6.2.3 

4.4 

Precluding  modem  technology 

3;6.2.4 

1^ 

"Umbrella"  clauses 

3:6.4 

tmm 

Environmental  constraints 

3:5.2.d 

4.7 

Traceability  of  requirements 

4.8 

Use  of  standards 

3:6.2.5 

5.1 

Uneven  level  of  requirements 

5.2 

Vague  requirements 

3:5.2.b.  6.1. b,  6.3, 
6.8.1 

5.3 

Underspecification,  overspecification 

6.1 

Use  of  defined  formats 

6.2 

Individual  format  issues 

6.3 

Numbering  of  requirements 

6.4 

Precedence 

6.5 

Dispersed  requirements 

7. 

Verification  of  requirements 

3:5.4.b 

8. 

Prioritisation  of  requirements 

3:4.1, 6.5,  7. 

9.1 

Language:  definition  and  use  of  terms 

9.2 

Language:  concurrency 

3:6.9.1 

9.3 

Language:  alternative  functions 

3:6.8.2 

10. 

Non-functional  requirements 

2:4.3.3,  3:6.10 

11. 

Security  issues 

3:6.11.3 

12. 

Electronic  form  of  specifications 

2:4.2.5,3:6.11 

13. 

Pre-release  of  specifications 

14.1 

Need  for  operational  information 

3:7.1 

14.2 

Need  for  quality  guidelines 

2:4.2.1,3:9. 

14.3 

Relationship  between  developer  and  customer 

3:7. 

3:7.1 

2:4.2.1.3:9. 
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